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Abstract 


A  key  goal  of  the  Navy’s  Digital  Warfare  Office  (DWO)  is  to  use  the  emerging  field  of 
big  data  analytics  to  tackle  numerous  challenges  facing  the  Navy.  DWO  asked  CNA  to 
examine  the  issue  of  Super  Hornet  (F/A-18E/F  strike  fighter)  readiness  and 
recommend  data-driven  solutions  that  leverage  underutilized  sensor  data.  CNA 
proposed  a  pilot  program  that  integrated  sensor  data  across  maintenance  levels  to 
expedite  repairs  of  aviation  parts.  The  five-month  pilot  program  began  on  July  10, 
2017,  at  the  Fleet  Readiness  Center  at  Oceana  in  Virginia  Beach,  Virginia,  and  was 
implemented  on  APG-65  and  APG-73  radars.  We  assessed  the  pilot  program  through 
several  metrics  and  found  that,  during  the  program,  repair  time  was  significantly 
decreased  and  repair  efficiency  increased.  Our  findings  suggest  that  sensor  data 
integration  across  maintenance  levels  may  considerably  improve  F/A-18  readiness. 


1 


CNA 


This  page  intentionally  left  blank. 


CNA 


Executive  Summary 


F/A-18  readiness 

The  Digital  Warfare  Office  (DWO)  was  established  in  2016  under  the  Chief  of  Naval 
Operations  to  lead  the  U.S.  Navy  in  transforming  from  the  industrial  to  the  digital 
age.  The  goal  of  the  DWO  is  to  enable  better  decision-making  across  all  of  the  Navy’s 
mission  and  functional  areas  by  using  the  multitude  of  data  that  the  Navy  collects 
more  effectively.  The  first  DWO  focus  area  is  F/A-18  readiness.  Currently,  more  than 
60  percent  of  the  Navy  and  U.S.  Marine  Corps  strike  fighters  are  unavailable  for 
missions— but  there  could  be  solutions  through  better  use  of  data.  The  naval  aviation 
community  collects  an  abundance  of  aircraft  sensor  data  that  have  the  potential  to 
expedite  repairs  and  hence  increase  readiness.  However,  this  trove  of  data  has  not 
yet  been  made  accessible  to  all  of  the  maintenance  staff  whose  work  could  be 
improved  through  its  use.  CNA  proposed  a  pilot  program— focused  on  the  Super 
Hornet  variant  of  the  F/A-18  strike  fighter  jet— to  test  a  possible  solution  to  the 
readiness  problem. 

Pilot  Program 

CNA  conducted  a  two-month  pilot  program  that  integrated  aircraft  sensor  data, 
known  as  Built-in-Test  (BIT)  data,  into  the  maintenance  repair  process.  BIT  data  are 
recorded  automatically  on  Super  Hornets  during  flight  when  a  fault  occurs  in  the 
aircraft.  Squadron  maintenance  crews  (organizational-level  (O-level)  maintainers)  rely 
heavily  on  BIT  data  to  troubleshoot  repairs  during  unscheduled  maintenance  on  the 
flight  line.  By  providing  intermediate-level  (I-level)  maintainers  access  to  the  BIT  data 
(which  they  did  not  previously  have),  the  root  cause  of  failure  could  potentially  be 
more  quickly  identified.  Faster  repairs  could  then  lead  to  an  increase  in  the  number 
of  mission-capable  aircraft.  Having  more  airplanes  “up”  (i.e.,  mission  capable)  and 
ready  to  complete  mission  sets  translates  to  better  readiness  levels. 

The  pilot  program  began  on  July  10,  2017,  at  the  Fleet  Readiness  Center  (FRC)  at 
Oceana  in  Virginia  Beach,  Virginia,  and  was  implemented  on  APG-65  and  APG-73 
radars.  Before  the  pilot  program,  parts  were  inducted  into  the  I-level  maintenance 
without  accompanying  sensor  data,  which  contain  information  on  the  part’s  failure. 
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The  pilot  program  implemented  a  process  improvement  by  requiring  that  BIT  data  be 
sent  to  I-level  maintenance,  along  with  the  part  in  need  of  repair,  to  assist 
maintainers  in  troubleshooting  the  problem  and  establishing  a  maintenance  plan  of 
action. 

CNA  Assessment 

To  assess  the  pilot  program,  CNA  constructed  and  tracked  several  metrics  pertaining 
to  repair  time  and  effectiveness.  The  two  metrics  of  primary  interest  were:  (1)  time  to 
reliably  replenish  (TRR)  and  (2)  number  of  parts  ordered  per  repair.  TRR  is  the  time 
required  to  troubleshoot  and  repair  failed  equipment  and  return  it  to  normal 
operating  conditions.  Our  analysis  found  that  when  I-level  maintainers  had  access  to 
and  used  BIT  data,  the  average  TRR  for  radar  repairs  was  reduced  by  45  percent 
compared  with  the  average  TRR  of  non-pilot  program  2017  data.  We  used  a  Monte 
Carlo  approach  to  assess  the  significance  of  these  results  to  determine  whether  the 
pilot  program’s  mean  TRR  occurred  by  chance  alone.  Our  confidence  in  the 
improvement  shown  in  TRR  during  the  pilot  program  is  96  percent.  The  pilot 
program  also  showed  that  the  average  number  of  parts  ordered  per  repair  was 
reduced  by  40  percent  when  BIT  data  were  available  and  used. 

These  results  suggest  that  the  integration  of  BIT  data  throughout  the  maintenance 
process  could  assist  maintainers  in  more  quickly  and  effectively  detecting  the  root 
causes  of  failures.  With  expedited  fault  detection,  fewer  unnecessary  parts  would 
need  to  be  ordered,  saving  time  and  money. 

Recommendations 

Based  on  our  findings,  we  recommend  that  the  Naval  Aviation  Enterprise  (NAE) 
leverage  sensor  data  integration  at  another  FRC  (e.g.,  Lemoore),  specifically  for 
repairs  of  a  system  that  has  a  large  impact  on  the  readiness  of  the  Super  Hornet  fleet 
(e.g.,  generator  control  units).  This  new  effort  should  include  training  for  both  O-  and 
I-level  maintainers  on  how  to  efficiently  and  effectively  provide  and  use  BIT  codes  for 
repairs.  Ideally,  the  effort  would  last  at  least  six  months,  so  that  the  impact  of  sensor 
data  integration  on  readiness  could  be  captured  for  analysis  and  evaluation. 

During  the  course  of  the  pilot  program,  we  discovered  a  few  issues  with 
infrastructure,  data  transfer,  data  interpretation,  and  data  rights  that  will  need  to  be 
resolved  before  the  full  benefits  of  BIT  data  can  be  realized.  If  BIT  data  were  to  be 
integrated  across  the  entire  fleet,  the  NAE  would  need  to  consider  the  following: 
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1.  How  to  incorporate  a  cyber-secure  electronic  transfer  of  sensor  data  from  the 
O-level  to  the  I-level 

2.  How  to  develop  robust  sensor  diagnostic  reasoners  that  could  be  updated  and 
matured  based  on  real  operational  maintenance  practices  and  findings 

3.  How  to  contract  for  the  rights  to  sensor  data  in  future  platforms  and  systems 

4.  What  storage  and  computing  infrastructure  would  be  necessary  to  house, 
query,  and  analyze  such  massive  datasets 
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Introduction 


The  Chief  of  Naval  Operations  (CNO)  established  the  Digital  Warfare  Office  (DWO)  in 
2016  to  develop  a  framework  that  prioritizes  data-driven  decision-making  in  an 
increasingly  informationalized  environment.  A  key  mission  of  the  DWO  is  to  use  the 
emerging  field  of  big  data  analytics  to  tackle  the  numerous  challenges  that  the  Navy 
faces.  The  Navy  already  collects  a  multitude  of  data,  but  this  data  is  often 
underutilized  and  not  shared  across  organizations.  DWO  is  leading  initiatives  to 
leverage  data  science  and  digital  technologies  to  produce  indicators  of  positive 
outcomes  that,  once  matured,  could  have  transformative  impacts  on  the  Navy’s 
competitiveness. 

The  first  challenge  the  DWO  approached  is  Super  Hornet  (F/A-18  E/F  strike  fighter) 
readiness.  Military  readiness  is  quantitatively  defined  as  the  number  of  resources 
available  in  individual  units  (e.g.,  strike  fighter  squadron)  versus  the  stated 
requirements  of  that  unit  [1].  Readiness  is  the  ability  of  military  units  to  carry  out 
their  assigned  missions  and  tasks;  it  can  be  affected  by  equipment,  training,  and 
availability  of  spare  parts,  among  other  factors  [2]. 

F/A-18  readiness  involves  a  large  number  of  systems,  organizations,  and  processes 
with  linkages  and  complex  interdependencies,  as  well  as  a  myriad  of  data  and 
metrics.  VADM  Paul  Grosklags,  Commander,  Naval  Air  Systems  Command  (NAVAIR), 
wrote  of  the  aviation  readiness  shortfall  that  “squadron  commanding  officers  are 
having  to  make  tradeoffs,  whether  it  is  a  training  mission  or  operational 
requirement,  on  a  daily  basis  because  they  do  not  have  the  required  number  of  Ready 
Basic  Aircraft  (RBA)1”  [4].  In  fact,  in  2016,  one  in  five  pilots  in  a  strike  fighter 
squadron  did  not  fly  enough  hours  to  meet  the  tactical  hard  deck2  (THD),  the 
minimum  number  of  hours  a  pilot  must  fly  per  month  for  safety  of  flight  [5].  Only  40 


1  Ready  Basic  Aircraft  is  defined  as  “the  minimum  configuration  required  to  conduct  day  or 
night  [Instrument  Meteorological  Conditions]  flight  operations  with  necessary  communications, 
[Identification  Friend  or  Foe],  navigations,  flight  and  safety  systems  required  by  applicable 
[Naval  Air  Training  and  Operating  Procedures  Standardization]  and  [Federal  Aviation 
Administration]  regulations.  This  aircraft  does  not  require  a  Functional  Check  Flight  and  does 
not  require  shipboard  operations  equipment  (no  outstanding  L  or  Z  [Equipment  Operational 
Capability]  discrepancies)”  [3].  An  aircraft  that  is  RBA  may  not  necessarily  be  mission  capable. 

2  The  THD  is  1 1  flight  hours  per  month. 
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percent  of  Super  Hornets  in  the  Navy’s  fleet  today  are  up  and  capable  of  carrying  out 
their  mission  sets.  Although  that  number  may  not  be  a  direct  measure  of  readiness, 
because  it  does  not  address  how  many  jets  the  Navy  is  required  to  have  up  to  meet 
its  operational  needs,  it  does  reflect  the  challenges  naval  aviation  is  facing  in 
managing  its  strike  fighter  inventory.  The  CNO  turned  to  DWO  to  develop  potential 
digital  solutions  to  this  problem. 

DWO  asked  CNA  to  examine  the  F/A-18  readiness  issue  and  provide  data-driven 
solutions  that  tap  into  the  abundance  of  underutilized  data  resources.  This  paper 
presents  the  results  of  CNA’s  pilot  program,  which  implemented  a  process 
improvement  in  aircraft  maintenance  to  expedite  repairs. 
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Opportunities  for  Improvement 


We  believe  that  readiness  could  be  improved  from  a  maintenance  perspective  by 
leveraging  a  dataset  in  naval  aviation  that  originates  from  sensors  in  the  aircraft: 
Built-in-Test  (BIT)  data.  With  minimal  additional  effort,  maintainers  could  take 
advantage  of  this  type  of  diagnostic  information  to  quickly  and  efficiently  identify 
the  root  cause  of  failure  and  thus  expedite  repairs  and  improve  readiness. 

What  are  BIT  data? 

BIT  data  originate  from  sensors  installed  on  Super  Hornet  aircraft  to  detect  and 
isolate  faults  down  to  the  subcomponent  level  [6].  Military  avionics  systems  rely 
heavily  on  BIT  data,  which  are  used  by  the  organizational  level  (O-level),  or  squadron 
level,  to  troubleshoot  repairs  during  unscheduled  maintenance  on  the  flight  line.  The 
data  are  recorded  on  the  aircraft’s  maintenance  card,  known  as  the  memory  unit 
(MU),  and  are  also  displayed  in  real-time  to  the  pilot.  When  a  BIT  code  appears  on  the 
pilot’s  display  panel  during  flight,  s/he  can  use  a  lookup  table  to  determine  which 
aircraft  system  may  have  failed  or  is  degraded.  BIT  codes  also  identify  failures  in 
critical  flight  subsystems  that  are  essential  to  the  aircraft’s  integrity  and 
airworthiness. 

Where  can  the  Naval  Aviation  Enterprise 
(NAE)  leverage  BIT  data? 

When  the  strike  fighter  pilot  returns  from  a  flight  with  a  failed  component,  s/he 
uploads  the  MU  into  Boeing-developed  software  known  as  the  F/A-18  Automated 
Maintenance  Environment  (FAME).  FAME  ingests  the  BIT  binary  data  recorded  in  the 
MU  during  pre-,  mid-,  and  post-flight  and  translates  the  data  to  human-readable  text. 
The  software  will  also  aggregate  frequent  BIT  codes  and  information  on  affected 
components  to  identify  important  trends.  After  uploading  the  MU  into  FAME,  the 
pilot  creates  a  Maintenance  Action  Form  (MAF)  with  his/her  notes  from  the  flight.  O- 
level  maintainers  at  the  flight  line  use  the  information  from  the  MAF,  with  the  aid  of 
the  BIT  data  from  FAME,  to  troubleshoot  problems.  The  BIT  data  help  maintainers  fix 
problems  quickly  so  that  the  downed  jet  can  return  to  its  mission  set. 
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When  the  failure  is  too  complex  and  requires  a  deeper  level  of  repair,  the  O-level 
maintainers  enter  additional  information  onto  the  MAF  and  send  the  affected  part  to 
intermediate  level  (I-level)  maintenance.  I-level  maintainers  have  access  to  the 
updated  MAF  but  are  never  provided  with  BIT  diagnostic  information.  An  example  of 
the  repair  flow  from  O-  to  I-level  for  a  radar  transmitter  is  illustrated  in  Figure  1. 

At  the  I-level,  as  shown  in  Figure  1,  maintainers  attempt  to  identify  failures  by 
connecting  the  component  to  a  Consolidated  Automated  Support  System  (CASS)  test 
bench  and  conducting  an  end-to-end  test.3  The  CASS  test  bench  is  not  foolproof;  our 
analysis  revealed  that  more  than  10  percent  of  the  time  the  CASS  test  bench  will  not 
be  able  to  detect  any  issues  with  the  part.  In  these  cases,  the  maintainer  will  record 
in  the  MAF  that  the  gripe  could  not  be  duplicated,  and  the  part  will  be  returned  to 
the  squadron  with  no  repairs  executed.  Alternatively,  the  CASS  bench  might  identify 
the  first  detectable  subcomponent  error  in  a  prescribed  sequence  of  tests  and  will 
not  continue  to  further  fault  test  until  that  error  is  mended.  The  error  might  not 
actually  be  the  root  cause  of  failure,  only  a  symptom  of  the  problem.  Consequently, 
once  the  subcomponent  is  replaced  or  fixed,  the  entire  part  will  be  connected  to  the 
test  bench  only  for  it  to  detect  additional  failures.  This  leads  to  multiple  iterations  of 
parts  orders  and  wasted  maintenance  man  hours  before  the  underlying  issue  is 
identified. 

We  hypothesize  that  if  the  BIT  data  were  shared  at  the  I-level,  the  root  cause  of 
failure  could  be  detected  faster.  Quicker  repairs  ultimately  lead  to  an  increase  in  the 
number  of  mission-capable  aircraft.  More  airplanes  up  and  ready  to  complete 
mission  sets  means  better  readiness  levels. 


3  An  end-to-end  test  on  the  CASS  bench  consists  of  a  sequence  of  coded  routines  that  test  all 
subcomponents  (known  as  shop  replaceable  assembly,  or  SRA)  in  the  broken  component 
(known  as  work  replaceable  assembly,  or  WRA). 
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Figure  1 .  Traditional  repair  flow  from  O-  to  l-level 
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Pilot  Program:  Use  of  BIT  Data 


CNA  focused  on  improving  readiness  through  better  use  of  data  in  the  maintenance 
process.  One  way  to  improve  readiness  through  maintenance  is  to  decrease  the  time 
it  takes  to  repair  a  gripe.  Reducing  time  to  repair  is  related  to  how  quickly  a 
maintainer  can  correctly  diagnose  the  problem  and  fix  it,  without  ordering 
unnecessary  parts.  For  example,  Navy  analysis  of  maintenance  data  showed  that 
technicians  had  trouble  diagnosing  faults  and  ended  up  ordering  many  parts  in 
multiple  maintenance  iterations.  As  a  result,  the  first  pass  yield— the  fraction  of  time 
that  problems  are  resolved  in  the  first  maintenance  pass— is  less  than  50  percent  [8]. 
The  number  of  maintenance  passes  directly  affects  the  time  it  takes  to  repair  a 
component.  Previous  CNA  analysis  showed  that  the  average  number  of  days  it  takes 
to  close  out  any  F/A-18E/F  component  repair  in  the  first  maintenance  pass  is  20  [7]. 
By  the  third  maintenance  pass,  the  number  of  days  increases  by  100. 

Pilot  program  set-up 

The  pilot  program  began  on  July  10,  2017,  at  the  Fleet  Readiness  Center  (FRC)  at 
Oceana  in  Virginia  Beach,  Virginia.  Data  were  collected  for  five  squadrons  (VFA-32, 
34,  83,  105,  and  106)  over  a  two-month  period,  with  the  last  collection  taking  place 
on  September  14,  2017.  Analysis  was  performed  only  on  APG-65  and  APG-73  radars. 
The  following  radar  components  were  repaired: 

•  Antennas 

•  Receivers 

•  Data  processors 

•  Transmitters 

•  Power  supplies 

Pilot  program  modifications  to  the  traditional  process 

The  pilot  program  required  two  major  modifications,  one  each  at  the  O-level  and  the 
I-level,  so  that  the  BIT  data  could  be  transferred  and  interpreted  correctly.  The  two 
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major  changes,  outlined  below,  altered  the  procedure  in  Figure  1  to  what  is  seen  in 
Figure  2  (shown  by  the  rectangles  with  the  red  font): 

1.  O-level  maintainers  were  required  to  send  a  printout  of  the  BIT  data  from 
FAME4  (for  example,  “B  codes  =  125,  432,  067,  O  codes  =  042,  A  codes  =  1255”) 
to  the  I-level  maintainers,  along  with  the  broken  part  and  the  MAF  (also  known 
as  a  work  order,  or  WO).  Parts  sent  without  the  BIT  data  were  not  designated  as 
pilot  program  repairs  by  the  I-level  and  were  not  utilized  in  the  analysis. 

2.  If  a  radar  component  was  sent  to  I-level  maintenance  with  the  FAME  printout, 
maintainers  were  required  to  enter  the  BIT  codes  into  a  diagnostic  reasoner 
developed  by  PMA-265.6  The  reasoner  provided  information  on  the  most 
probable  subcomponent(s)  that  needed  to  be  repaired.  Maintainers  would  then 
utilize  the  translated  BIT  data,  along  with  the  CASS  bench  test  results,  to 
determine  the  best  course  of  action  (i.e.,  what  subcomponent  to  test  and 
repair). 

a.  When  I-level  maintainers  updated  the  MAF,  they  were  asked  to  mark  pilot 
program  repairs  with  either  *SM65*  for  APG-65  repairs  or  *SM73*7  for  APG- 
73  repairs  in  the  “system  reasoner”  field  of  the  Naval  Aviation  Logistics 
Command  Management  Information  System  (NALCOMIS). 

This  effort  did  not  dictate  an  exact  repair  process  to  the  I-level  maintainers.  Instead, 
the  pilot  program  required  maintainers  only  to  consider  both  the  BIT  data  and  CASS 
test  bench  recommendations.  This  way,  I-level  maintainers  at  Oceana,  with  the  help 
of  the  maintenance  master  chief  petty  officer,  could  construct  a  repair  plan  that  was 
easiest  for  them  to  implement.  After  speaking  with  the  maintainers  and  looking  at 
the  repair  comments  in  the  data,  we  determined  that  there  were  three  main  courses 
of  action  taken  with  the  BIT  data,  as  shown  in  Figure  2  by  the  diamonds  with  the  blue 
font: 

1.  Trust  the  BIT  data:  Some  of  the  BIT  codes  are  well  understood  by  maintainers. 
They  trust  these  codes  implicitly  and  use  them  to  diagnose  and  fix  parts.  For 


4  Ideally,  O-level  maintainers  would  extract  the  BIT  data  from  FAME  and  share  it  with  the  I-level 
maintainers  electronically  via  cloud  computing  or  some  sort  of  SharePoint  site.  This  was  not 
done  for  the  pilot  program  due  to  time  and  resource  restrictions. 

5  B  =  Operator  or  Initiated  BIT,  0  =  Start-up/Power-on  BIT,  A  =  Accumulated/Periodic  BIT 

6  The  reasoner  is  in  the  form  of  an  Excel  macro  that  has  the  ability  to  translate  the  list  of  BIT 
codes  entered  by  the  maintainer  into  insightful  output.  The  output  is  continuously  updated 
based  on  real-world  maintenance  data. 

7  *SM65*  and  *SM73*  were  chosen  by  the  master  chief  and  maintainers  at  Oceana. 
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example,  BIT  104  for  the  APG-73  radar  is  always  known  to  indicate  a  fault  in 
the  radar  transmitter,  specifically  in  the  low  voltage  circuits. 

2.  Trust  the  BIT  data  after  verification:  Some  of  the  BIT  codes  are  not  well 
understood,  so  a  flight  line  check8  would  be  performed  to  verify  that  the  BIT 
code  was  indicating  the  correct  failure.  The  BIT  code  was  accepted  only  after  it 
passed  the  flight  line  test.  If  a  flight  line  check  failed,  the  gripe  was  addressed 
with  the  CASS  bench  recommendations. 

3.  Do  not  trust  the  BIT  data:  In  these  cases,  the  BIT  code  was  not  well  understood, 
and  the  maintainer,  for  reasons  unknown  to  the  CNA  analysts,  did  not  trust  or 
use  the  BIT  data  in  the  repair  plan.  In  these  cases,  the  repair  was  conducted 
without  the  aid  of  BIT  data. 

Data  collection 

All  of  the  data  used  in  our  analysis  were  derived  from  NALCOMIS.  An  aviation 
technician  first  class  at  Oceana  extracted  the  necessary  fields  for  the  analysts  and 
compiled  them  in  a  Microsoft  Access  database.  The  database  held  information  on  all 
radar  repairs  from  January  1,  2017,  to  November  14,  2017,  including  those  from  the 
pilot  program  (marked  by  *SM65*  or  *SM73*).  We  received  data  updates  every  two  to 
three  weeks  for  the  duration  of  the  pilot  program. 

Data  collection  caveat 

During  the  pilot  program,  three  out  of  five  high-powered  CASS  test  benches  were  out 
of  service.  This  situation  delayed  repair  times  for  radar  transmitters  and  power 
supplies,  which  demand  high  voltages  and  can  be  tested  only  with  a  high-powered 
CASS  bench.  As  a  result,  we  omitted  these  two  components  from  the  data  collected 
for  both  the  pilot  program  and  baseline9  all  together.  Out  of  112  pilot  program  data 
points  collected,  42  (37.5  percent)  were  removed,  leaving  us  with  70  repairs  to  use  in 
our  analysis.  Out  of  350  baseline  data  points,  137  (39  percent)  were  removed,  leaving 
us  with  213  repairs  in  this  dataset. 


8  The  part  would  be  sent  to  the  O-level,  placed  back  on  the  aircraft,  and  tested  at  the  flight  line 
to  determine  if  the  gripe  was  corrected  with  the  aid  of  the  BIT  data. 

9  The  baseline  dataset  is  what  the  pilot  program  is  compared  to  (non-pilot  program  data)  and  is 
described  later  in  the  Data  Selection  section. 
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Figure  2.  Pilot  program  repair  flow  from  O-  to  l-level 
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Pilot  program  metrics 

Maintenance  repair  time 

We  measured  maintenance  repair  time  with  the  following  three  metrics: 

1.  Time  to  Reliably  Replenish  (TRR):  The  time  required  to  troubleshoot  and  repair 
failed  equipment  and  return  it  to  normal  operating  conditions.  It  is  measured 
as  the  delta  between  the  time  a  part  is  received  at  the  FRC  (i.e.,  the  induction 
date)  and  when  it  leaves  the  FRC  with  a  “complete”  tag.  It  includes  the  time  the 
part  is  on  the  shelf  at  the  FRC  waiting  for  inspection,  subcomponents,  and/or 
repair,  but  does  not  include  any  supply  lead  time.1011 

2.  Elapsed  Maintenance  Time  (EMT):  “Time,  in  hours  and  tenths,  that  maintenance 
was  being  performed  on  a  job”  [9]. 

3.  Maintenance  Man  Hours  (MMH)12:  “The  total  number  of  accumulated  direct 
labor  hours  (in  hours  and  tenths)  expended  in  performing  a  maintenance 
action”  [9]. 

We  used  TRR  as  the  primary  repair  time  metric  because  it  provides  the  best  insight 
into  how  quickly  the  root  cause  of  failure  was  diagnosed.  EMT  and  MMH  might  not 
have  as  much  of  a  direct  impact  in  reducing  pilot  program  repairs  because  all  repairs 
are  still  required  to  be  verified  on  the  CASS  test  bench  [10].  Nevertheless,  we 
measured  these  secondary  metrics  to  ensure  consistency  of  increase/decrease  in 
overall  repair  time. 

Efficiency 

TRR  is  affected  by  the  efficiency  of  repairs,  which  is  determined  in  part  by  how 
successful  a  maintainer  is  at  diagnosing  and  fixing  the  cause  of  a  failure  without 


10  Supply  lead  times  can  be  long  if  a  part  is  shipped  from  a  deployed  squadron  to  FRC  Oceana 
for  repair. 

11  TRR  includes  all  repairs,  even  if  a  gripe  could  not  be  duplicated  (malfunction  code  799). 
Including  799  repairs  (-2.5%  of  the  data)  did  not  change  the  results. 

12  EMT  and  MMH  are  related  but  not  equivalent.  From  [9]:  “if  five  men  complete  a  job  in  2.0 
hours  of  continuous  work,  the  EMT=2.0  hours  and  the  man  hours=10.0.” 
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having  to  order  parts  in  multiple  iterations.  The  following  two  metrics  were  chosen 
as  a  proxy  to  quantify  efficiency: 

1.  Order  iterations:  The  number  of  maintenance  passes  for  a  repair  in  which  one 
or  more  orders  were  made 

2.  Number  of  parts  ordered:  The  total  number  of  parts  ordered  in  a  repair 
Details  of  these  metrics  are  described  in  Appendix  A:  Metrics. 
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Data  Selection 


The  results  of  this  study  are  based  on  two  datasets:  the  baseline  dataset  and  the  pilot 
program  dataset.  The  baseline  dataset  includes  information  on  all  the  APG-65/73 
repairs  inducted  at  Oceana  from  January  1,  2017,  through  July  10,  2017.  There  are 
213  repairs  in  the  baseline  dataset,  none  of  which  were  part  of  the  pilot  program. 
The  pilot  program  dataset  includes  information  on  APG-65/73  repairs  inducted  at 
Oceana  from  July  10,  2017,  through  November  20,  2017,  that  were  marked  as  part  of 
the  pilot  program  and  where  BIT  data  were  called  out  as  being  available  to  the  I-level 
maintainer  in  the  MAF.  There  are  39  repairs  documented  in  the  pilot  program 
dataset.  We  describe  the  selection  of  data  in  more  detail  below. 

There  are  70  repairs  tagged  with  a  “*SM65*”  or  “*SM73*”  in  NALCOMIS,  indicating 
that  these  70  repairs  were  supposed  to  be  part  of  the  pilot  program.  However,  not 
every  repair  tagged  as  part  of  the  pilot  program  actually  included  I-level  access  to  BIT 
data  during  the  repair  process.  This  could  occur  if  a  part  were  inducted  to  the  I-level 
as  part  of  the  pilot  program  (resulting  in  a  *SM*  being  placed  in  the  MAF),  but  when 
the  I-level  maintainer  conducts  the  repair,  which  can  happen  several  days  after 
induction,  s/he  realizes  that  the  BIT  data  that  was  sent  from  O-level  are  incorrect. 

To  determine  whether  repairs  were  inappropriately  tagged  as  part  of  the  pilot,  we 
examined  the  notes  provided  for  each  repair  by  the  I-level  maintainer.  These  notes 
were  included  in  every  MAF  in  the  “corrective  action”  block  and  were  unformatted 
free-text.  Out  of  the  70  repairs,  only  39  explicitly  indicated  that  BIT  data  were 
available  for  the  repair.  Below  are  examples  of  corrective  actions  listed  for  those  39 
repairs: 

1.  “REMOVED  AND  REPLACED  4 A1  IDENTIFIED  VIA  BOA13  CODE  311  ON  MAF  AND 
IN  CASS  TEST  NUMBER  3106  IN  PROGRAM  3525041...” 

2.  “RESEATED  MULTIPLE  SRA'S  IDENTIFIED  VIA  BOA  CODE  161...” 

3.  “BOA  DATA  RECEIVED  INDICATED  A  PROBLEM  WITH  2A12...” 


13  BOA  is  another  term  for  BIT  data. 
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4.  “RDP  RECEIVED  WITH  BOA  CODES:  O:  RDP  054,435.  A:  054/01  WITH  SMART 
CALLOUTS  BEING  4A2,  4A6,  AND  4A15...” 

In  cases  such  as  the  above  examples,  it  was  clear  the  appropriate  data  were  provided 
to  the  I-level  maintainer.  However,  many  corrective  actions  indicated  that  the  BIT 
codes  provided  by  the  O-level  pointed  to  the  incorrect  component.  Examples  of  such 
corrective  actions  included: 

1.  “BIT  DATA  POINTED  TO  A  PROBLEM  WITH  THE  TRANSMITTER  NOT  THE 
RADAR  RECEIVER...” 

2.  “BOA  DATA  POINTED  TO  A  PROBLEM  WITH  THE  XMTR  NOT  THE  RR. 

In  both  of  the  above  examples,  radar  receivers  were  under  repair,  but  the  BIT  data 
provided  were  for  radar  transmitters.  The  FAME  software,  from  which  the  BIT  data 
were  extracted  by  O-level  maintenance,  contains  many  different  BIT  datasets  for 
different  components  and  aircrafts.  What  likely  happened  in  such  cases  is  that  the  O- 
level  simply  provided  the  BIT  data  for  a  different  component14  than  what  was  actually 
sent  to  I-level  for  repair.  For  the  scope  of  this  pilot  demonstration,  only  minimal 
training  was  provided  to  the  O-level  on  how  to  extract  BIT  data,  which  is  the  likely 
cause  for  such  errors.  More  comprehensive  training  could  help  maintainers  fully 
leverage  these  datasets. 

For  other  repairs  tagged  as  part  of  the  pilot,  comments  expressly  noted  that  BIT  data 
were  not  considered  during  the  repair  (for  reasons  unknown  to  CNA).  An  example: 

3.  “BORESIGHTED,  REALIGNED  AND  REMOVED  AND  REPLACED  THE  LISTED 
PARTS  IN  ACCORDANCE  WITH  AW-640LO-740-030...  PILOT  PROGRAM  NOT 
USED.” 

Many  repairs  contained  corrective  actions  that  made  no  mention  of  the  pilot  program 
or  any  receipt  or  use  of  BIT  data.  When  there  was  no  indication  of  BIT  data  utilization 
in  a  repair,  it  was  removed  from  the  pilot  program  dataset. 

Our  final  pilot  program  sample  contains  information  on  39  repairs  where  there  was 
explicit  indication  that  BIT  data  were  available  for  the  I-level  maintainer  to  leverage. 
Even  in  these  cases,  the  BIT  data  were  not  always  deemed  useful.  In  the  following 
section  we  show  the  results  of  our  analysis  on  this  dataset.  Analysis  on  the  full  70 
data  points  is  provided  in  Appendix  B:  Unfiltered  Data. 


14  BIT  codes  are  provided  in  FAME  for  every  component  that  failed  a  routine  diagnostic  test 
during  a  Super  Hornet’s  flight.  Some  BIT  codes,  and  their  time  sequences,  indicate  failure,  while 
others  are  cautions  or  warnings  of  degrades. 
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Pilot  Program  Results 


In  this  section,  we  present  the  results  of  the  pilot  program.  Table  1  summarizes  the 
comparison  of  pilot  program  results  with  the  baseline  data. 

Table  1 .  Comparison  of  pilot  program  results  with  baseline  data 


Maintenance  Repair  Time 

Metric 

Pilot 

Program 

Baseline 

Percent 

Difference 

Mean  TRR  (Days) 

10.17 

18.38 

44.6% 

Mean  EMT  (Hours) 

17.76 

21.88 

18.8% 

Mean  MMH  (Hours) 

43.47 

53.73 

19.1% 

Efficiency 

Metric 

Pilot 

Program 

Baseline 

Percent 

Difference 

Mean  #  of  Order  Iterations 

1.15 

1.73 

33.6% 

Mean  #  of  Parts  Ordered 

3.05 

5.39 

43.4% 

Our  analysis  found  that  the  primary  repair  time  metric,  mean  TRR,  was  significantly 
reduced  when  maintainers  at  the  I-level  had  access  to  BIT  data.  In  the  pilot  program 
dataset,  the  average  TRR  was  reduced  by  45  percent,  down  to  10.17  days,  compared 
with  the  baseline  of  18.38  days  per  repair.  The  improvement  in  the  mean  TRR 
stemmed  from  the  pilot  program’s  reduction  of  very  long  repair  times.  This  is  shown 
in  Figure  3,  where  the  cumulative  distribution  function  (CDFs)15  of  the  pilot  program 


15  A  CDF  is  the  probability  that  a  metric  is  less  than/equal  to  a  particular  value.  In  Figure  3,  the 
CDF  is  the  probability  that  the  TRR  is  less  than/equal  to  the  number  of  days  indicated  on  the  x- 
axis. 
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is  compared  with  the  baseline.  For  the  pilot  dataset,  all  repairs  took  less  than  50  days 
to  complete,  whereas  in  the  baseline  dataset  the  TRR  for  many  repairs  extended 
beyond  90  days.  Incorporating  BIT  data  into  I-level  repairs  eliminated  a  great  fraction 
of  instances  of  long  repair  times.  The  pilot  program  also  shows  a  decrease  in  the 
hands-on  repair  times,  EMT  and  MMH,  from  the  baseline  by  approximately  20 
percent. 

Figure  3.  CDF  comparison  of  TRR  between  the  pilot  program  dataset  and  baseline 
dataset 


Time  To  Reliably  Replenish  (days) 


The  reduction  in  TRR  for  the  pilot  program  was  partially  driven  by  a  reduction  in  the 
number  of  parts  being  ordered  per  repair,  as  well  as  a  reduction  in  the  number  of 
order  iterations  per  repair.  In  the  2017  baseline  data,  the  average  number  of  parts 
ordered  per  radar  repair  was  5.4,  while  during  the  pilot,  the  average  number  of  parts 
ordered  was  reduced  to  3.0  (a  more  than  40  percent  reduction  in  average  number  of 
parts  ordered  per  repair).  These  results  suggest  that  the  integration  of  BIT  data 
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throughout  the  maintenance  process  could  assist  maintainers  in  more  quickly  and 
effectively  detecting  the  root  causes  of  failures.  Expediting  the  identification  of  root 
cause  of  failures  translates  to  fewer  unnecessary  parts  ordered,  a  reduction  in  the 
money  spent  on  unnecessary  parts,  and  a  reduction  of  unnecessary  logistic  delays. 
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Validity  of  Results 


When  validating  our  results,  we  focused  on  the  primary  metric  of  interest,  mean  TRR, 
and  used  a  Monte  Carlo  approach  to  assess  the  significance  of  the  results  of  the  pilot 
program.  With  this  approach,  we  were  able  to  determine  that  the  probability  of 
obtaining  a  mean  TRR  that  is  less  than  or  equal  to  the  pilot  program  mean  TRR  of 
10.17  days  from  a  randomly  drawn  sample  is  four  percent. 

The  Monte  Carlo  approach  we  used  draws  a  random  sample  of  39  consecutive16 
repairs  from  the  baseline  dataset.  After  the  sample  is  chosen,  the  primary  metric 
(mean  TRR)  is  calculated  for  the  sample.  The  process  is  repeated  a  million  times  to 
create  a  distribution  of  the  mean  TRR,  which  is  then  compared  with  the  same  metric 
from  the  pilot  program  data.17  Figure  4  shows  the  distribution  of  mean  TRRs  for  the 
collection  of  the  million  draws  of  39-point  sample,  with  the  mean  TRR  for  the  pilot 
program  (10.17  days)  indicated  by  the  blue  dotted  line.  The  probability  of  obtaining  a 
mean  TRR  that  is  less  than  or  equal  to  the  pilot  program  mean  TRR  of  10.17  days 
from  a  randomly  drawn  sample  is  four  percent.  In  other  words,  the  baseline  data  has 
only  a  four  percent  chance  of  reproducing  the  pilot’s  TRR  results  by  chance. 


16  Historical  maintenance  data  show  variability  among  repair  times  for  different  quarters  of  the 
year.  For  consistency  with  the  pilot  program,  we  drew  samples  of  consecutive  repairs  instead 
of  randomly  chosen  repairs. 

17  A  diagram  of  this  process  can  be  found  in  Appendix  C:  Flow  Diagram  of  Monte  Carlo 
Approach 
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Figure  4.  Monte  Carlo  comparison  of  baseline  program  versus  pilot  program  data 
mean  TRRa 


Mean  Time  To  Reliably  Replenish  (days) 


The  pilot  program  mean  TRR  shown  in  blue  dotted  line. 


Why  Monte  Carlo? 

Statistical  tests  come  in  two  broad  classes:  parametric  and  nonparametric  analyses. 
Nonparametric  tests  are  useful  when  the  data  do  not  follow  a  specific  distribution. 
The  TRR  distributions  in  Figure  5  show  that  the  baseline  distribution  cannot  be 
entirely  represented  by  a  standard  distribution  such  as  Weibull  or  Lognormal. 
Although  not  shown,  the  same  is  true  for  the  distribution  of  TRR  in  the  pilot 
program.  This  means  that  nonparametric  methods  likely  provide  more  accurate 
statistics  when  comparing  the  pilot  program  and  baseline  distributions.  The  problem 
with  using  standard  nonparametric  tests  for  statistical  significance,  such  as 
Kolmogorov-Smirnov  or  Anderson  Darling,  is  the  variability  found  across  tests.  For 
example,  the  Anderson  Darling  test  is  more  accurate  when  the  distributions  in 
question  have  longer  tails,  while  the  Kolmogorov-Smirnov  test  is  more  sensitive  to 
deviations  near  the  center  of  the  distribution  than  at  the  tails.  For  these  reasons,  this 
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analysis  used  a  Monte  Carlo  approach  to  determine  how  likely  or  unlikely  the  pilot 
program  results  came  from  the  baseline  data  distribution. 


Figure  5.  Illustration  of  Weibull  and  lognormal  CDF  fit  to  the  baseline  distribution 
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Conclusions  and  Recommendations 


The  pilot  program  was  able  to  successfully  improve  the  time  and  efficiency  of  repairs 
by  integrating  BIT  data  throughout  the  maintenance  process.  Sensor  data  may 
therefore  considerably  improve  readiness  for  Super  Hornets  by  expediting  repairs  of 
parts  that  may  be  keeping  jets  in  a  non-mission  capable  status. 

Because  of  the  limited  sample  size,  short  duration  of  this  pilot  program,  and  system 
used  (i.e.,  APG-65  and  73),  we  recommend  that  NAE  expand  the  effort  to  integrate 
sensor  data  at  another  FRC  (e.g.,  Lemoore)  on  a  system  that  has  a  large  impact  on  the 
readiness  of  the  Super  Hornet  fleet.  A  candidate  system  is  the  generator  control 
units,  which  have  been  a  top  degrader  for  Super  Hornets  for  some  time  [11].  This 
new  effort  should  include  training  for  both  O-  and  I-level  maintainers  on  efficiently 
and  effectively  providing  and  using  BIT  codes  for  repairs.  This  effort  would  ensure 
that  the  right  data  are  accessible  and  useful  to  I-level  maintainers  to  maximize  the 
outcome.  Ideally,  this  effort  would  last  at  least  six  months  to  allow  the  impact  of 
sensor  data  integration  on  readiness  to  be  captured  for  analysis  and  evaluation. 

The  pilot  program  revealed  a  few  outstanding  issues  that  need  to  be  addressed  to 
maximize  the  benefits  of  BIT  data.  They  included  infrastructure,  data  transfer,  data 
interpretation,  and  data  rights.  If  the  Navy  decides  to  integrate  BIT  data  not  just  for 
one  system  in  one  location  but  rather  across  the  fleet,  the  NAE  must  consider:  1.) 
how  to  incorporate  a  cyber-secure  electronic  transfer  of  sensor  data  from  O-  to  I- 
levels,  2.)  how  to  develop  more  robust  sensor  diagnostic  reasoners  that  can  be 
updated  and  matured  based  on  real  operational  maintenance  practices  and  findings, 
3.)  how  to  contract  for  the  rights  to  sensor  data  in  the  future,  and  4.)  what  storage 
and  computing  infrastructure  is  necessary  to  house,  query,  and  analyze  such  massive 
datasets. 
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Appendix  A:  Metrics 

We  describe  how  each  metric  is  calculated. 
Table  2.  Description  of  metric  calculations 


Metric 

Calculation 

TRR 

Completed  Date  -  Induction  Date 

MH 

Directly  reported  in  NALCOMIS 

EMT 

Directly  reported  in  NALCOMIS 

Order 

Iterations 

The  number  of  orders  per  repair  is  stated  in  NALCOMIS. 

An  order  iteration  accounts  for  all  orders  made  on  the  same 
day. 

If  the  order  was  made  the  following  day,  that  is  an  additional 
order  iteration. 

Parts 

Reported  in  NALCOMIS  and  added  for  all  subcomponents 
ordered  during  a  repair  (job  control  number) 
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Appendix  B:  Unfiltered  Data 


As  noted  in  the  Data  Selection  section,  70  repairs  were  tagged  with  a  “*SM*”, 
indicating  the  I-level  maintainer  should  have  had  access  to  and  used  BIT  data  in  the 
repair  process.  Only  in  39  of  the  70  repairs  was  it  clear  BIT  data  were  actually  used, 
and  the  results  for  those  39  repairs  are  presented  in  the  body  of  this  work.  In  this 
appendix,  we  present  the  analysis  on  all  70  data  points  for  completeness.  Table  3 
summarizes  the  metrics  for  the  unfiltered  70  data  points,  along  with  the  same 
metrics  for  the  baseline  data  and  the  pilot  program  data  described  in  the  Pilot 
Program  Results  section. 

Table  3.  Comparison  of  metrics  for  the  filtered  pilot  program  data  with  baseline 
data 


Metric 

Baseline 

Unfiltered 

Percent 

Pll0t  rvff 

_  Difference 

Program 

Pilot 

Percent 

Program: 

Difference 

BIT  used 

Mean  TRR  (Days) 

18.38 

12.5  32.0% 

10.17  44.6% 

Mean  #  of  Order 
Iterations 

1.73 

1.51  12.8% 

1.15  33.6% 

Mean  #  of  Parts 
Ordered 

5.39 

4.29  20.6% 

3.05  43.4% 

Metric 

Baseline 

Unfiltered 

Pilot  Percent 

Program  Difference 

Pilot 

Program:  Percent 

BIT  used  Difference 

Mean  EMT  (Hours) 

21.88 

23.93  -9.4% 

17.76  18.8% 

Mean  MMH  (Hours) 

53.73 

64.03  -19.2% 

43.47  19.1% 
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In  the  unfiltered  pilot  program  dataset,  the  average  TRR  was  reduced  by  32  percent, 
down  to  12.5  days  from  the  baseline  of  18.38  days  per  repair.  When  we  isolated  and 
analyzed  only  the  repairs  that  we  are  confident  had  access  to  BIT  data— the  BIT  used 
dataset  in  Table  3— the  reduction  in  average  TRR  was  nearly  45  percent,  down  to  10.2 
days  from  the  baseline  of  18.38  days.  Though  both  the  unfiltered  pilot  program  and 
pilot  filtered  (i.e.,  pilot  BIT  used)  datasets  demonstrated  the  ability  to  lower  the 
number  of  part  orders  and  reduce  TRR,  our  analysis  of  the  EMT  and  MMH  metrics  is 
more  difficult  to  interpret.  The  full  pilot  dataset  revealed  longer  hours  spent  turning 
wrenches  through  the  EMT  and  MMH  metric  than  the  baseline  EMH  and  MMH. 
However,  of  the  70  repairs  in  the  full  pilot  dataset,  those  that  had  the  longest  EMTs 
and  MMHs  did  not  indicate  the  use  of  BIT  data  to  detect  the  root  cause  of  failure, 
while  the  pilot  BIT  incorporated  data  showed  a  20  percent  reduction  in  both  metrics. 
It  is  not  clear  why  the  maintainers  might  have  needed  more  hands-on  time  to  fix 
parts  in  the  unfiltered  pilot  program  dataset. 
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Appendix  C:  Flow  Diagram  of  Monte 
Carlo  Approach 


Figure  6.  Monte  Carlo  approach  diagram 
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